Hacking VMAF and VMAF NEG: Vulnerability to Different Preprocessing Methods

M. Siniukov, A. Antsiferova, D. Kulikov, and D. Vatolin

Contact us: aantsiferova@graphics.cs.msu.ru, and video@compression.ru

Abstract

Video quality measurement plays a critical role in the development of video processing applications. In this paper, we show how popular quality metrics VMAF and its tuning-resistant version VMAF NEG can be artificially increased by video preprocessing. We propose a pipeline for tuning parameters of processing algorithms which allows to increase VMAF by up to 218.8%.

A subjective comparison of preprocessed videos showed that with the majority of methods visual quality drops down or stays unchanged. We show that VMAF NEG scores can also be increased by some preprocessing methods by up to 21.9%.

(PDF, 14.4 MB)

Key Features

  • Increase VMAF by up to 218.8% and VMAF NEG by up to 21.9%
  • Comparation of 8 different preprocessing methods
  • Results verification on encoded streams
  • Powered by Subjectify.us

Examples

VMAF preprocessing example
After preprocessing VMAF was increased by 181.22% and visual quality drops
VMAF NEG preprocessing example
After preprocessing VMAF NEG was increased by 13.66% and visual quality does not change

Below you can see VMAF gain in procents for the best preprocessing methods. CLAHE gives the largest gain for VMAF.

Best preprocessing methods for VMAF Best preprocessing methods for VMAF

(ZIP, 15.2 MB)

Cite Us

@inproceedings{10.1145/3508259.3508272,
    author = {Siniukov, Maksim and Antsiferova, Anastasia and Kulikov, Dmitriy and Vatolin, Dmitriy},
    title = {Hacking VMAF and VMAF NEG: Vulnerability to Different Preprocessing Methods},
    year = {2021},
    isbn = {9781450384162},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/3508259.3508272},
    doi = {10.1145/3508259.3508272},
    abstract = {Video quality measurement plays a critical role in the development of video processing applications. In this paper, we show how popular quality metrics VMAF and its tuning-resistant version VMAF NEG can be artificially increased by video preprocessing. We propose a pipeline for tuning parameters of processing algorithms which allows to increase VMAF by up to 218.8%. A subjective comparison of preprocessed videos showed that with the majority of methods visual quality drops down or stays unchanged. We show that VMAF NEG scores can also be increased by some preprocessing methods by up to 21.9%.},
    booktitle = {2021 4th Artificial Intelligence and Cloud Computing Conference},
    pages = {89–96},
    numpages = {8},
    keywords = {video quality measurement, VMAF, video quality, video codecs comparisons, quality improvement, objective full-reference metric, codecs tuning},
    location = {Kyoto, Japan},
    series = {AICCC '21}
}

Contact Us

Interested in this research? Contact us: aantsiferova@graphics.cs.msu.ru, and video@compression.ru

See Also

Written on April 5, 2022